Papers with Textual Inference
Natural Language Inference with Mixed Effects (2020.starsem-1)
Copied to clipboard
| Challenge: | aggregating raw annotations to a single label is problematic due to disagreement among annotators. |
| Approach: | They propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise. |
| Outcome: | The proposed method improves performance over models that do not incorporate such effects. |
Compositional Semantics and Inference System for Temporal Order based on Japanese CCG (2022.acl-srw)
Copied to clipboard
| Challenge: | a system for temporal order in Japanese has not been developed for linguistic inference involving temporal expressions. |
| Approach: | They propose a Japanese NLI system that considers temporal order in Japanese . they use axioms for temporal relations and automated theorem provers to perform inference involving temporal orders. |
| Outcome: | The proposed system outperforms logic-based systems and current deep learning models on Japanese datasets. |
Annotation Artifacts in Natural Language Inference Data (N18-2)
Copied to clipboard
| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |
PerspectroScope: A Window to the World of Diverse Perspectives (P19-3)
Copied to clipboard
| Challenge: | PerspectroScope is a web-based system that lets users query a discussion-worthy natural language claim . |
| Approach: | They propose a web-based system which lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim. |
| Outcome: | The proposed system lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim. |
Logical Inferences with Comparatives and Generalized Quantifiers (2020.acl-srw)
Copied to clipboard
| Challenge: | Comparative constructions pose a challenge in Natural Language Inference (NLI) Comparatives interact with quantifiers, numerals, and lexical antonyms, but a logical inference system for comparatives has not been developed for the task. |
| Approach: | They propose a compositional semantics system that maps comparative constructions to semantic representations via combinatory categorial grammar parsers and integrates it with an automated theorem proving system. |
| Outcome: | The proposed system outperforms previous logic-based systems and deep learning models on three NLI datasets. |
PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are prone to factually inconsistent statements, known as hallucinations. |
| Approach: | They propose to train a specialized model that detects inconsistencies over text prefixes to improve generation faithfulness by 5-14 F1 points. |
| Outcome: | The proposed model outperforms baseline models by 5-14 F1 points in prefix-level entailment. |
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (N18-1)
Copied to clipboard
| Challenge: | et al., 1996, show that many of the most actively studied problems in NLP depend in large part on natural language understanding (NLU). |
| Approach: | They propose a dataset for machine learning that uses ten different genres of English to evaluate sentences for their meanings. |
| Outcome: | The multi-genre natural language inference corpus is one of the largest available for natural language understanding. |
Scoring Lexical Entailment with a Supervised Directional Similarity Network (P18-2)
Copied to clipboard
| Challenge: | Existing word embeddings that use supervision only improve the embeddable word embeds of words with annotated lexical relations. |
| Approach: | They propose a supervised directional similarity network for learning task-specific transformation functions on top of general-purpose word embeddings. |
| Outcome: | The proposed model outperforms existing models on the HyperLex dataset on a directional graded lexical entailment task by 25%. |
MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain (2021.acl-short)
Copied to clipboard
| Challenge: | a large number of crowdworker-constructed datasets have been used to conduct natural language inference (NLI) on unstructured, domainspecific texts such as patient notes, pathology reports, and scientific papers. |
| Approach: | They investigate whether MedNLI contains lexical and syntactic annotation artifacts associated with annotation process that allow hypothesis-only classifiers to achieve better-than-random performance. |
| Outcome: | The proposed model outperforms a majority-class baseline model on a physician-annotated dataset with premises extracted from clinical notes. |
Structured Alignment Networks for Matching Sentences (D18-1)
Copied to clipboard
| Challenge: | Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity. |
| Approach: | They propose a model of structured alignments between sentences to compare two sentences by matching their latent structures. |
| Outcome: | The proposed model is differentiable and trained only on the matching objective. |
Lessons from Natural Language Inference in the Clinical Domain (D18-1)
Copied to clipboard
| Challenge: | State of the art models with deep neural networks lack generalization capabilities in specialized domains where training data is limited. |
| Approach: | They propose a dataset annotated by doctors performing a natural language inference task grounded in the medical history of patients. |
| Outcome: | The proposed model outperforms existing models in the clinical domain by incorporating domain knowledge from external data and lexical sources. |
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Recent advances on abstractive summarization have led to fluent summaries, but factual errors in generated summary still severely limit their use in practice. |
| Approach: | They evaluate summaries produced by state-of-the-art models via crowdsourcing and show that factual errors occur frequently. |
| Outcome: | The proposed models can detect errors and reduce them by reranking alternative summaries. |
Harry Potter and the Action Prediction Challenge from Natural Language (N19-1)
Copied to clipboard
| Challenge: | Using textual descriptions of scenes, we explore the challenge of action prediction from textual description. |
| Approach: | They propose a testbed to approximate whether text inference can be used to predict upcoming actions from textual descriptions of scenes. |
| Outcome: | The proposed model performs best for frequent actions and large scene descriptions, but logistic regression fails on infrequent actions. |
A Simple Three-Step Approach for the Automatic Detection of Exaggerated Statements in Health Science News (2021.eacl-main)
Copied to clipboard
| Challenge: | Exaggerations in health news can have tremendous adverse effects on the lifestyle of the common masses who feed themselves mostly on such news instead of the source scientific publication. |
| Approach: | They propose a three-step approach that extracts relation phrases from a scientific paper and then classifies the strength of the relationship phrase extracted. |
| Outcome: | The proposed approach outperforms baseline models that compare state-of-the-art embedding of the statement pairs through a binary classifier or recast the problem as a textual entailment task. |
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Social and behavioral determinants of health (SDOH) play a significant role in shaping health outcomes, and extracting these determinant from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. |
| Approach: | They propose a dataset that extracts social and behavioral determinants from clinical notes and uses them to form a natural language inference task. |
| Outcome: | The proposed dataset is based on publicly available notes and is more challenging than standard NLI benchmarks. |
Evaluating Factuality in Generation with Dependency-level Entailment (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text generation models produce text that is factually inconsistent with information in the input. |
| Approach: | They propose a new formulation of entailment that decomposes it at the level of dependency arcs. |
| Outcome: | The proposed model can identify factual errors better than sentence-level methods and question generation methods while localizing the erroneous parts of the generation. |
ViNLI: A Vietnamese Corpus for Studies on Open-Domain Natural Language Inference (2022.coling-1)
Copied to clipboard
| Challenge: | a large-scale corpus is needed for studies on natural language inference (NLI) for Vietnamese, which can be considered a low-resource language. |
| Approach: | They propose a corpus for evaluating Vietnamese natural language inference models . they use a human-annotated corpus extracted from more than 800 online news articles . |
| Outcome: | The ViNLI corpus is created and evaluated with a strict process of quality control . the best system performance is still far from human performance (a 14.20% gap in accuracy). |
MorphNLI: A Stepwise Approach to Natural Language Inference Using Text Morphing (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models fail to capture important semantic features of logic such as monotonicity and negation. |
| Approach: | They propose a modular step-by-step approach to natural language inference . they use a language model to generate edits to incrementally transform the premise into the hypothesis . |
| Outcome: | The proposed method outperforms baseline models in realistic cross-domain settings with improvements up to 12.6% (relative). |
Does Putting a Linguist in the Loop Improve NLU Data Collection? (2021.findings-emnlp)
Copied to clipboard
Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, Samuel R. Bowman
| Challenge: | Many datasets for training and evaluating natural language understanding (NLU) models contain systematic artifacts that are identified only after data collection is complete. |
| Approach: | They propose to have linguists identify artifacts and gaps in the data and communicate with non-expert crowdworkers to adjust task instructions and incentives. |
| Outcome: | The proposed protocol does not increase accuracy on out-of-domain test sets, and adds a chatroom does not. |
Conditional Natural Language Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | False positive: widely-used feature-attribution explanation models are not suitable for finding conditions when sentences are long and are written independently. |
| Approach: | They propose a task of conditional natural language inference to explain sentence pairs that provide contradictory (different) information for different conditions. |
| Outcome: | The proposed model outperforms the full cross-encoder and other baselines in extracting conditions while maintaining comparable accuracy. |
Posing Fair Generalization Tasks for Natural Language Inference (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for deep learning semantics rely on naturalistic corpora, but they often fail to support the kind of generalization we are asking for. |
| Approach: | They define and motivate a formal notion of fairness for evaluations of deep learning models for semantics . they then apply it to natural language inference by constructing challenging but provably fair artificial datasets based on the results . |
| Outcome: | The proposed evaluations show that standard neural models fail to generalize in the required ways and even these models do not solve the task perfectly. |
Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic (2024.emnlp-main)
Copied to clipboard
Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme
| Challenge: | Recent language models allow structured reasoning with text, but lack of a clear protocol for discerning entailment causes noisy datasets and limited performance gains. |
| Approach: | They propose a consistent approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference. |
| Outcome: | The proposed approach has higher internal consistency than prior decompositional entailment datasets and significantly improves proof quality and accuracy. |
Baselines and Test Data for Cross-Lingual Inference (L18-1)
Copied to clipboard
| Challenge: | Recent research on textual entailment is limited to English, but it is expanding to other languages. |
| Approach: | They propose to extend the research in SNLI-style natural language inference toward multilingual evaluation by using cross-lingual word embeddings and machine translation. |
| Outcome: | The proposed system scores an average accuracy of just over 75%, but it is not perfect. |
Word Attribute Prediction Enhanced by Lexical Entailment Tasks (2020.lrec-1)
Copied to clipboard
| Challenge: | a semantic attribute is associated with a designated dimension in attribute-based vector representations . semantic attributes are created by psychological experimental settings involving human annotators . a conceptual attribute of a concept dictates a specific semantic aspect of the concept . |
| Approach: | They propose a two-stage neural network architecture that fine-tunes attribute representations by employing supervised entailment tasks. |
| Outcome: | The proposed method improves performance of semantic/visual similarity/relatedness evaluation tasks. |
An Analysis of Natural Language Inference Benchmarks through the Lens of Negation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for natural language inference ignore negations and can make inferences that are difficult to make. |
| Approach: | They propose a new benchmark for natural language inference in which negation plays a critical role. |
| Outcome: | The proposed benchmarks show that negation plays a critical role in inference judgments. |
Uncertain Natural Language Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Existing categorical bin assignments for NLI are not effective for entailment inference. |
| Approach: | They propose a refinement of Natural Language Inference that targets direct prediction of subjective probability assessments. |
| Outcome: | The proposed model can be used in pre-training and correlates well with humans. |
New Datasets for Automatic Detection of Textual Entailment and of Contradictions between Sentences in French (2024.lrec-main)
Copied to clipboard
| Challenge: | DACCORD is a dataset for automatic detection of contradictions between sentences . it is the first dataset exclusively dedicated to this task . |
| Approach: | They introduce DACCORD, a dataset in French for automatic detection of contradictions between sentences. |
| Outcome: | The proposed datasets are more challenging than existing datasets for the mainstream task in French. |
Uncovering Agendas: A Novel French & English Dataset for Agenda Detection on Social Media (2024.lrec-main)
Copied to clipboard
| Challenge: | a social media analysis of online influence campaigns can reveal the sources of agenda setting . annotated data is limited or nonexistent, but there are methods to detect agenda control . |
| Approach: | They propose a method for detecting instances of agenda control through social media . they use a modest corpus of tweets centered on the 2022 french presidential election . |
| Outcome: | The proposed method overcomes the requirement for large annotated training dataset. |
Entailed Between the Lines: Incorporating Implication into NLI (2025.acl-long)
Copied to clipboard
Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya, Alex Fabrikant
| Challenge: | True Emotions, social cues, insults, and a myriad of other messages are conveyed implicitly, often even more so than explicitly. |
| Approach: | They propose a dataset to help LLMs understand implied entailment . |
| Outcome: | The proposed dataset enables LLMs to understand implied entailment and can generalize this understanding across datasets and domains. |